Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

LAMAReg's robust mode uses two-stage registration for improved accuracy:
1. Stage 1: Parcellation-based coarse alignment (primary warpfield)
2. Stage 2: Direct image refinement (secondary warpfield)

This commit adds --secondary-warpfield and --inverse-secondary-warpfield
arguments to all LAMAReg registration commands to prevent automatic
composition, which causes accuracy loss.

Changes:
- 02_proc-dwi.sh: Add secondary warpfields for T1w→DWI registration
- 02_proc-func.sh: Add secondary warpfields for func→T1nativepro registration
- 02_proc-flair.sh: Add secondary warpfields for FLAIR→T1nativepro registration
- 03_MPC.sh: Add secondary warpfields for microstructural→FreeSurfer registration
- 03_MPC-SWM.sh: Add secondary warpfields for microstructural→T1nativepro registration

Output naming convention:
- {prefix}1Warp.nii.gz: Primary warpfield (parcellation-based)
- {prefix}2Warp.nii.gz: Secondary warpfield (image-based refinement)
- {prefix}1InverseWarp.nii.gz: Primary inverse warpfield
- {prefix}2InverseWarp.nii.gz: Secondary inverse warpfield

Transformation application order (ANTs applies right-to-left):
- Forward: -t secondary_warp -t primary_warp -t affine
- Inverse: -t affine -t primary_inverse -t secondary_inverse

This ensures no composition loss and maintains full registration accuracy.

Related: #156

Comprehensive documentation covering:
- Why secondary warpfields are necessary
- Changes made to each module
- Output file naming conventions
- Transformation application order
- Benefits and testing checklist

Related: #156

Created systematic unit tests for all LAMAReg registration modules:
- test_dwi_registration.sh: DWI to T1w registration
- test_func_registration.sh: Functional MRI registration
- test_flair_registration.sh: FLAIR registration
- test_mpc_registration.sh: MPC FreeSurfer registration
- test_mpc_swm_registration.sh: MPC SWM registration

Test Framework Features:
- Validates LAMAReg command syntax and arguments
- Checks for all required output files (including secondary warpfields)
- Verifies file integrity and NIfTI headers
- Analyzes DICE scores from QC CSV files
- Tests transformation chain correctness
- Validates two-stage robust registration outputs

Supporting Scripts:
- run_all_tests.sh: Master test runner for full suite
- test_common.sh: Shared test functions and utilities
- validate_outputs.sh: Standalone output validation tool
- README.md: Test suite overview and requirements
- USAGE.md: Comprehensive usage guide with examples

Test Validation:
✓ LAMAReg installation and CLI availability
✓ ANTs tools present
✓ Command syntax (--secondary-warpfield support)
✓ Primary warpfields (1Warp.nii.gz, 1InverseWarp.nii.gz)
✓ Secondary warpfields (2Warp.nii.gz, 2InverseWarp.nii.gz)
✓ Parcellation outputs (fixed, moving, registered)
✓ QC CSV with DICE scores
✓ Transformation application order
✓ File integrity and headers

Tests can run with or without real data:
- Syntax validation works without inputs
- Full validation requires test images
- Configurable DICE score thresholds
- Detailed logging and reporting

Related: #156

Comprehensive summary document covering:
- All 4 commits with detailed changes
- Technical implementation details
- Repository structure overview
- Test suite coverage
- Breaking changes documentation
- Performance considerations
- Next steps and validation checklist
- Complete statistics and references

This document serves as the authoritative reference for the entire
LAMAReg integration project.

Related: #156

Comprehensive guide explaining:
- What test data is actually needed (processed files, not raw BIDS)
- Where to find files in micapipe derivatives
- Specific files needed for each test module
- Step-by-step setup instructions
- Quick setup script for automating data preparation
- Common questions and troubleshooting

This helps users understand they need intermediate processed files
from micapipe runs, not raw subject data.

Related: #156

Automated script to copy test data from HCP subject (sub-211215)
to test directory. Handles:
- DWI registration test files
- Functional MRI test files
- FLAIR test files (if available)
- MPC test files with FreeSurfer conversion
- Summary and next steps instructions

Configured for bb-compxg-01 server paths.

Related: #156

- Temporarily disable 'set -e' when running individual tests
- Capture exit codes to prevent premature termination
- Allow all tests to run even if one fails
- Properly track passed/failed test counts

- Add TEST_SUBDIRS array mapping tests to data subdirectories
- Pass correct paths (e.g., test_data/dwi, test_data/func) to each test
- Skip tests gracefully when data directory doesn't exist
- Track and report skipped tests in summary
- Fixes issue where only DWI test was running

- Use actual filename: space-dwi_desc-T1w_nativepro_SyN.nii.gz for DWI
- Use space-fsnative_T1w.nii.gz instead of FreeSurfer orig.mgz
- Skip functional test (no func data in this HCP subject)
- Remove mri_convert dependency

- Use find command to locate func brain in nested directory structure
- HCP has func data in desc-se_task-rest_dir-LR_run-1_bold/volumetric/
- Update diagnostic script to use find for func files

- Add execute_lamareg() function to test_common.sh
- Update all test scripts to actually run LAMAReg registration
- Add real-time output with tee to log file
- Show execution status and timing
- Remove syntax-only validation, now tests run full registration

- Replace eval with direct command execution
- Remove string quoting issues that prevented execution
- Add proper exit code checking with PIPESTATUS
- Add verbose logging

- set -e was causing the test to exit silently on any error
- This prevented actual test execution and LAMAReg from running
- Now test will continue and show actual errors

- Replace all echo -e with printf for better shell compatibility
- Some shells don't properly handle echo -e with escape sequences
- This was causing silent test failures

… use plain echo

- Remove set -e to avoid silent exits on errors
- Remove all color code variables (RED, GREEN, BLUE, etc.)
- Replace all echo -e with plain echo commands
- Use $((var + 1)) arithmetic instead of ((var++))
- Simplified test structure for better reliability
- Tested locally with fake data to verify basic execution works

…verbosity

- Remove set -e to avoid silent exits
- Remove all color codes and echo -e calls
- Replace ((var++)) with var=$((var + 1))
- Add verbose output showing when LAMAReg is ACTUALLY running vs being skipped
- Add clear warnings when input files are missing
- Add execution time tracking for LAMAReg
- Add output file verification after LAMAReg runs
- Add LAMAReg version display
- Make it obvious to users why tests complete fast (missing input files)

This fixes the issue where tests appeared to run but were actually
skipping LAMAReg execution due to missing input files.

CRITICAL BUG FIXED:
- LAMAReg uses SynthSeg which requires 3D anatomical images
- Previous code used 4D FOD (spherical harmonics) which fails
- Error: 'ValueError: operands could not broadcast (4,) (3,)'

CHANGES:
- test_dwi_registration.sh: Auto-detect 3D vs 4D images
- Prefer dwi_b0.nii.gz > dwi_FA.nii.gz > dwi_fod_vol0.nii.gz
- Warn clearly when 4D image is used (will fail)
- Check dimensionality with fslinfo if available

- setup_hcp_test_data.sh: Extract 3D reference from DWI data
- Priority: b0 > FA > FOD first volume
- Use fslroi to extract vol0 from 4D FOD if needed

DOCUMENTATION:
- BUG_DWI_4D_FOD.md: Complete bug report and fix recommendations
- Documents why this fails and how to fix main pipeline

NEXT STEP:
- Main pipeline (02_proc-dwi.sh) needs same fix
- Should use dwi_b0 instead of fod for --fixed parameter

- Update proc_dwi_transformations() to use LAMAReg method
- Update proc_func_transformations() to use LAMAReg method
- Read from exported $reg variable (set in processing scripts)
- Add 'registrationMethod' field to JSON for clarity
- Update log messages to show registration method used

This completes the TODO item: 'update JSON metadata' for 02_proc-dwi.sh